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REDUCED STATE SEQUENCE ESTIMATION SIGNAL DETECTION METHOD 



FIELD OF THE INVENTION 

The present invention relates to signal detection and in particular to detection of signals 
using trellis systems. 
5 BACKGROUND OF THE INVENTION 

Communication channels generally carry information in the form of symbols which 
represent one or more binary bits. Communication channels generally add noise to the symbols 
transmitted on them. In addition, communication channels smear out the signals transmitted on 
them, in a process referred to as inter-symbol interference (ISI). Receivers generally include an 
10 equalizer that determines the transmitted symbols based on the symbols received from the 
channel. 

Fig. 1 is a schematic block diagram of a transmission system 20. A transmitter 22 
transmits a sequence of symbols {X n } over a channel 24, to a receiver 26. Each symbol X n 
may take on one value from a set of values referred to as an alphabet or constellation. The 
15 number of values in the constellation is referred to as D. Channel 24 distorts the sequence of 
symbols {X n } in a manner which is generally modeled by a convolution with a function {fjj 
having K+l non-zero terms, such that receiver 26 receives symbols {r n } which have the 
modeled form of: 
K 

r n = Hfk x n-k+™n 
k=0 

20 in which w n represents noise added by channel 24. The number of terms in function 

{fjj is (K+l) and is referred to as the length of channel 24. An equalizer 28 generates 
respective estimates {x n } (referred to also as detections) of the transmitted symbols {X n } 
from the received symbols {r n }, based on a model {f^} of channel 24. 

There exist various types of equalizers which perform their task (referred to as 

25 equalization), using different methods. These methods include linear equalization (LE), 
decision-feedback equalization (DFE) and maximum likelihood sequence estimation (MLSE). 
The MLSE method chooses from among a set of possible transmitted sequences a single 
sequence which is most likely to have been transmitted in view of the received symbols and a 
model of the channel. In a typical MLSE application, a state machine (referred to as a trellis) is 

30 created in which each possible transmitted sequence is represented by a path going through a 
plurality of states. Typically, each state represents a short sequence of a predetermined number 
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of recently transmitted symbols. Each transition between states (referred to as a branch of the 
trellis) represents a single transmitted symbol. While MLSE has much lower error rates than 
the other methods, its computation and space complexity is enormous and therefore in many 
cases its use is substantially impossible. 
5 A document titled "Reduced-State Sequence Estimation with Set Partitioning and 

Decision Feedback", M.V. Eyuboglu, S.U.H. Qureshi, IEEE Trans. Commun., Vol. 36, No. 1, 
January 1988, suggests a simplification of MLSE referred to as reduced state sequence 
estimation (RSSE). In the RSSE method, set partitioning methods are used in order to reduce 
the number of states in the trellis. The RSSE method introduces a subset trellis with subset 

10 states, in which each subset state represents one or more states of the MLSE. 

SUMMARY OF THE INVENTION 
An aspect of some embodiments of the present invention relates to an equalizer which 
performs symbol sequence estimation using a trellis in which states and transitions are defined 
based on a multi dimensional (MD) system model. In the MD model, a plurality of transmitted 

15 symbols are viewed as a single MD-symbol. Particularly, in some embodiments of the 
invention, the branches of the MD subset trellis represent one or more multi-dimensional 
symbols which are formed of a plurality of transmitted symbols. Using the MD model, set 
partitioning methods may be used on the MD-constellation to generate an MD-subset trellis 
which has lower computation and/or space complexity than can be achieved using the prior art 

20 for predetermined performance requirements. Alternatively, better performance than prior art 
methods may be achieved with comparable computation and/or space complexity. 

In some embodiments of the invention, the states of the trellis comprise subsets which 
represent a plurality of MD-symbol value permutations. 

In some embodiments of the invention, an MD-ML (maximal likelihood) trellis is 

25 defined based on the MD model. The MDML trellis is searched to provide a detected symbol 
sequence for a received sample sequence. These embodiments are referred to as MD-MLSE 
(multi dimensional - maximal likelihood sequence estimator). 

In some embodiments of the invention, an MD-subset trellis is defined based on the 
MD model and a partitioning of the MD constellation of the model into MD subsets. The MD- 

30 subset trellis is searched to provide a detected symbol sequence for a received sample 
sequence. These embodiments are referred to as MD-RSSE (Multi-dimensional - reduced state 
sequence estimator). In some embodiments of the invention, an MD-RSSE trellis is 
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constructed in a manner which optimizes a tradeoff, for example, between performance and 
complexity. 

In some embodiments of the invention, the transmission is indifferent to the MD 
system model of the equalizer. Alternatively, the transmitter optimizes the transmitted 
5 sequence based on knowledge of the structure of the MD-RSSE or other decoding method 
which is used by a receiving end. 

In some embodiments of the invention, the performance and/or complexity of the MD- 
RSSE are analyzed and accordingly the method is tailored to a specific channel and/or to 
specific performance requirements. 
10 A potential advantage of some embodiments of the invention is that the equalization 

method can be analyzed. 

In some embodiments of the invention, the sequence estimation methods described 
herein are used as a cost calculation method for selecting an optimal transmission method. In 
one example, a transmission sequence is selected from among a plurality of transmission 
15 sequences, using an MD-RSSE, instead of, for example, a reduced Viterbi method, as used in 
the art. Substantially any desired cost function may be used, for example a function modeling 
expected channel noise statistics. It should be noted that the receiver may be any receiver and 
does not necessarily use the equalizer methods described herein. 

There is thus provided in accordance with an exemplary embodiment of the invention, 
20 a method of detecting symbols in a signal transfer system, comprising: 

transferring symbols on a channel which may distort the symbols; 
representing the transferred symbols by multi-dimensional (MD) symbols, which each 
includes a plurality of transferred symbols; 

providing a trellis which has states which are defined by one or more MD symbols, and 
25 which has branches which correspond to respective MD symbols; 
receiving a plurality of samples; and 

calculating estimate values of one or more transferred symbols based on the received 
samples and the provided trellis. 

Optionally, the MD symbols comprise a plurality of transferred symbols which take on 
30 values from identical value sets. Alternatively or additionally, at least one of the states of the 
provided trellis comprises a subset state which represents a plurality of different MD-symbol 
values. Optionally, the trellis comprises states which represent values of a predetermined 
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number of MD symbols and wherein providing the trellis comprises generating the trellis by 
partitioning the possible values of each of the MD symbols in the states of the trellis into one 
or more subsets and wherein the states of the trellis comprise all the permutations of the 
subsets of the values of the multi-dimensional symbols. Optionally, partitioning the possible 
5 values of each of the MD symbols in the states of the trellis into one or more subsets comprises 
grouping the possible values of different MD symbols of the trellis states into different 
numbers of subsets. Alternatively, partitioning the possible values of each of the MD symbols 
in the states of the trellis into one or more subsets comprises partitioning the possible values of 
the MD symbols which include earlier transmitted symbols into no more subsets than the 

10 possible values of the MD symbols which were transmitted later were partitioned into. 

In an exemplary embodiment of the invention, partitioning the possible values of each 
of the MD symbols in the states of the trellis into one or more subsets comprises partitioning 
the possible values of later transmitted symbols into subsets of which all the members are 
included in a single subset of earlier transmitted symbols. 

15 In an exemplary embodiment of the invention, the method comprises generating a 

model of the channel and wherein providing the trellis comprises providing a trellis generated 
responsive to the channel model. Alternatively or additionally, wherein providing the trellis 
comprises providing a predetermined trellis. Alternatively or additionally, providing the trellis 
comprises providing a trellis determined responsive to the received samples. Alternatively or 

20 additionally providing the trellis comprises optimizing a trellis for a particular transmission 
situation. Alternatively or additionally providing the trellis comprises: 
generating a plurality of possible trellises; 

determining for each of the possible trellises one or more of a performance or a 
complexity parameter; and 
25 selecting one of said trellises responsive to said determining. 

In an exemplary embodiment of the invention, an equalizer, comprises: 
a input channel for receiving received signals; 

a processor for applying a MD trellis methods as described herein; and 
an output channel for providing an estimate of symbols received on said input channel, 
30 based on said applying. Optionally, the equalizer is incorporated in a cellular telephone. 

BRIEF DESCRIPTION OF FIGURES 
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Particular non-limiting embodiments of the invention will be described with reference 
to the following description of embodiments in conjunction with the figures. Identical 
structures, elements or parts which appear in more than one figure are preferably labeled with a 
same or similar number in all the figures in which they appear, in which: 
5 Fig. 1 is a schematic block diagram of a transmission system known in the art; 

Fig. 2 is a flowchart illustrating a method for producing a trellis for symbol decoding, 
in accordance with an exemplary embodiment of the present invention; 

Fig. 3 illustrates a multi-dimensional view of a communication system, in accordance 
with an embodiment of the present invention; 
10 Fig. 4 is a schematic illustration of producing a trellis for an exemplary channel, in 

accordance with an exemplary embodiment of the present invention; 

Fig. 5 is a schematic block diagram of a data structure managed by an equalizer, in 
accordance with an exemplary embodiment of the present invention; 

Fig. 6 is a flowchart of the acts performed by an equalizer in decoding received signals, 
15 in accordance with an embodiment of the present invention; 

Fig. 7 is a schematic block diagram of a multi-input multi-output (MIMO) symbol 
detector, known in the art; 

Fig. 8 is a schematic map of a four-way set partitioning for the rectangular 2D 16 QAM 
signal set, in accordance with an embodiment of the present invention; and 
20 Fig. 9 is a schematic illustration of four possible set partitionings for various channels, 

in accordance with an exemplary embodiment of the present invention. 

DETAILED DESCRIPTION OF EMBODIMENTS 
Fig. 2 is a flowchart illustrating a method for producing a trellis for symbol detection, 
in accordance with an exemplary embodiment of the present invention. Given a 
25 communication channel 24, a one dimensional model of the channel is generated (50) using 
any method known in the art, for example by transmitting a known test sequence through the 
channel 24 and determining the effect of the channel on the test sequence. In some 
embodiments of the invention, the model is the form of an FIR (finite impulse response) filter 
Amotion, with K+l taps (i.e., coefficients). 
30 For the following description an exemplary channel modeled by the following function: 

/o = i 

/ = [/l J2J3J4] = [0.1,-0.1,02,0.8] (1) 
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and having a constellation (alphabet) of two (D=2) possible symbol values {-1,+1 }, is used. 

Based on the channel model, a number of symbols (K) to be included in states (denoted 
p n ) of the trellis, is chosen. Optionally, the chosen number of symbols (K) is equal to the 
length of the channel, i.e., the number of non-negligible coefficients in the model of the 
5 channel (excluding fo). In the example of equation (1), the number of symbols (K) included in 
states p n of the trellis is K=4. 

Alternatively to generating the model and selecting K based on the model, an estimate 
of the length of the model is used. Thus, the trellis may be generated before a model of the 
channel is generated. This may be used in cases in which the channel changes over time. 
10 At any time n, the K symbols recently transmitted by transmitter 22 may take on any of 

the permutations p n = [x n _ x ,x n _ 2 ,... 9 x n _ K ] of length K of the symbols in the constellation of 

channel 24. Permutations p n are referred to herein as trellis states. In the MLSE method known 
in the art, a maximum likelihood (ML) trellis is built, based on the states p n . The states p n and 
transitions x n uniquely define the next state p n +i- In the RSSE method known in the art, a 

15 subset trellis is built, based on subset-states t n , which together with transitions x n uniquely 
define the next subset-state t n +j. The subset-state t n of the subset-trellis is equal to a set of 
states of the ML-trellis. 

In some embodiments of the invention, a multi-dimensional (MD) model for system 20 
is created (52), as shown in Fig. 3 which illustrates a multi-dimensional view of a 

20 communication system 30, in accordance with an embodiment of the present invention. The 
creation of the MD model may be explicit or implicit. In a multi-dimensional approach to 
system 20, although transmitter 22 transmits a sequence of single dimension symbols {x n } 
over channel 24, to receiver 26, the transmitter is viewed as transmitting for MD-symbols 
{ x n }. Each MD-symbol x n takes the following form: 




x n\f+M-2 



25 

in relation to the symbols X n which were actually transmitted. 
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10 



Each multi-dimensional symbol x n may take on one value from a set of values referred 
to as an MD-alphabet or MD-constellation. The number of values in the constellation is equal 
to D M , as each of the M symbols of the MD-symbols may take on any one of D possible 
values. 

In some embodiments of the invention, a number of MD symbols (L) to be included in 
the states of the created trellis is chosen (54). In some embodiments of the invention, 



-[II- 



Optionally, LM = K 5 when K is divisible by M. Alternatively, when K is not 



divisible by M, one or more padding 0 taps are concatenated to the filter. 

In some embodiments of the invention, a multi-dimension distortion model of channel 
24 is created as a set of L+l matrices {F^}, which take on the form: 



JkM -I 



_ jfifcA/-JLHI JkM-M 42 



JkM 



where ffc are the taps of the one dimensional channel model. 

For the simplicity of the following description, the received output samples are viewed 
as MD-output-samples, in which each MD output symbol r n includes a plurality of received 
symbols r n . In an exemplary embodiment of the invention, the following equation relates 
15 between the MD-output-samples and the one-dimensional output samples r n : 



r njVf+jV/-I 
r nM+l 



Thus, the relation between the transmitted MD-input symbols x n and the MD-received 
output samples r n is modeled as: 



In = £**iEn-* + Itin 
k-0 
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where w n is a proper noise vector. 

Optionally, the complexity and/or performance requirements of the equalizer are 
assessed (55) and accordingly a trellis type is chosen. 

In some embodiments of the invention, for example when a high performance is 
5 required, a multi-dimensional maximal likelihood (MD-ML) trellis is constructed (64). The 
MD-ML trellis states p n are defined by the last L MD-symbols transmitted by the transmitter 
22: 

The MD-ML trellis states, together with the currently transmitted MD-symbol x n , (which 
10 defines an MD-ML trellis transition) uniquely define the next MD-ML state, p n +i. Each 
transition of the MD-ML trellis is equal to M consecutive transitions in one-dimensional ML- 
trellises known in the art, such as described in G.D. Forney, Jr., "Lower Bounds on Error 
Probability in the Presence of Large Intersymbol Interference", IEEE Trans. Commun., 
February 1972, the disclosure of which is incorporated herein by reference. This trellis can be 
15 searched (66), for example, using a Viterbi algorithm, in order to detect the transmitted MD- 
symbols x n based on the received MD symbols r n . Generally, the searching of the MD-ML 

trellis achieves optimal performance based on the maximum likelihood criteria but in some 
cases requires high computation and/or complexity. 

Alternatively or additionally, an MD-subset trellis is constructed. The MD-subset trellis 

20 comprises subset-states t n which are defined by grouping together a subset of one or more 
states p n of the MD-ML trellis. In some embodiments of the invention, the MD constellation 
of each MD symbol x n _ k (k~l..L) in state p n is partitioned (56) into an M-dimensional subset, 
i.e., all the possible values of each MD symbol x H _ k are partitioned into one or more subsets. 
The partitioning may be performed, for example, as described in a document titled "Trellis- 

25 coded modulation with multidimensional constellation", L.F. Wei, IEEE Trans. Inform. 
Theory, Vol. 33, No. 4, July 1987, the disclosure of which is incorporated herein by reference. 
The constellation of each multi-dimensional symbol x n _ k is partitioned into Jfc subsets, in a set 
partitioning denoted D(k). The number Jfc of subsets into which the symbols are grouped may 
be anywhere between one and D M . Optionally, the subsets of the MD symbol x n _ k in the 
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partitioning Q(k) are ordered and a specific one of the subsets is referred to by a n _ k (k) which 

take on an integer between 0 and J^-l. 

Optionally, for at least one of the M-dimensional symbol subsets, the number of 
subsets is smaller than D M . In some embodiments of the invention, the numbers are non- 
5 increasing (i.e. J\ >~ J2 >= • ■ >= Jl )■ Additionally or alternatively, the multi-dimensional 
partitioning Cl{k) is a further partition of the subsets of Q(k+1) for each k between 1 and L-l. 

In some embodiments of the invention, subset states t n are created (58) from all the 
permutations of the M-dimensional subsets: 

t n = [an-i(l); Gfi-2(2). - - . , ^-£(1)] 

10 The number of states of the trellis is therefore J\j^ k=l,2,...,L. In some embodiments of the 

k 

invention, when all Jfc = D M for k=l,...,L, the MD-subset trellis is equivalent to the MD-ML 
trellis. 

Alternatively to partitioning (56) the MD constellation of each MD symbol x n _ k into an 

M-dimensional subset and creating the subset states t n from the permutations of the M- 
15 dimensional subsets, any other method for partitioning the MD-ML trellis states Pn into subset 
states t n , is used. Preferably, the partitioning is performed such that the subset-states t n 
together with any subset a n of the current transmitted MD-symbol x n uniquely define a next 
subset state t n +i. 

In some embodiments of the invention, the subset partitioning is performed in a 
20 manner which optimizes one or more performance parameters of the equalizer performing the 
symbol detection. Optionally, the one or more optimized performance parameters comprise 
one or more of a probability of error rate, a detection error rate, a computation complexity and 
a space complexity. In some embodiments of the invention, the subset grouping is performed 
in a manner which minimizes a detection error rate of the equalizer given a maximal 
25 computation complexity and/or a maximal space complexity. Alternatively, any other tradeoff 
between the computation and/or space complexity and the detection error rate of the equalizer, 
is used. 

In some embodiments of the invention, a plurality of potential subset partitionings of 
the MD symbols are prepared. For each of the partitionings one or more performance 
30 parameters are calculated and the grouping with the best performance parameters according to 
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a predetermined optimization criteria, is chosen. Various single- and multi-dimensional search 
methods, as known in the art, may be used to optimize the parameters and/or to select which 
potential subset partitionings to generate. 

In some embodiments of the invention, the subset grouping includes determining one 

5 or more of the number of symbols M included in an MD symbol, the number of subset-states 
t n in the MD-subset trellis, the number of subsets Jfc for each MD symbol and/or the actual 
MD-set partitioning Q(k) for each MD-symbol. 

In some embodiments of the invention, the number of symbols M included in an MD 
symbol is chosen arbitrarily, e.g., M=2. If the optimization does not achieve minimal required 

1 0 values for the optimized parameters, a greater or smaller value of M may be chosen. 

In some embodiments of the invention, the number of subset-states (t n ) of the MD- 
subset trellis is chosen based on computational and/or space complexity considerations and the 
MD-ML trellis states (pn) included in each of the groups are chosen based on an error rate 
parameter. Optionally, also the number of subsets J] is chosen based on complexity 

15 considerations as J\ affects the number of distinct branches 48 in the trellis. Alternatively or 
additionally, the number of subset-states (t n ) of the MD-subset trellis is chosen so as to 
achieve a desired error rate. In some embodiments of the invention, the number of subset- 
states (t n ) of the MD-subset trellis is chosen such that the error rate is only slightly higher than 
the error rate of a trellis in which each trellis state has a separate state (i.e. the MD-ML trellis). 

20 Optionally, for a given number of subset-states (t n ) of the trellis the error rate is determined for 
a best subset-state grouping. If the error rate is not substantially higher than the error rate for a 
trellis in which each trellis state has a separate state, subset-state groupings are prepared for a 
smaller number of subset-states (t^ of the trellis. This procedure is repeated until a subset 
grouping with a desired error rate is achieved. Optionally, the number of subset-states (t n ) of 

25 the trellis is reduced by a factor of two in each repetition. Alternatively, any other reduction 
rate and/or method is used in progressing between repetitions. 

In some embodiments of the invention, the error rate or the probability of error is 
determined or approximated as described hereinbelow or as described in US provisional patent 
application 60/159,410, filed October 14, 1999, or as described in Lior Ophir, "Multi- 

30 Dimensional Reduced State Sequence Estimation", thesis submitted towards the degree of 
"Master of Science in Electrical Engineering" in Tel Aviv university, October 1999, the 
disclosure of which documents is incorporated herein by reference. 
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Fig. 9 is a schematic illustration of four possible set partitionings for various channels, 
in accordance with an exemplary embodiment of the present invention. In some embodiments 
of the invention the partitioning of the of multi-dimensional sets is performed considering the 
matrix Fq in hand. Due to the fact that the multi-dimensional constellation undergoes a linear 
transformation, which is governed by the value of F0, it is possible to design the set 
partitioning according to this matrix. In some embodiments of the invention the goal of 
maximizing d'mjnCE'O is desired. For example, for 



1 0.9 
0 1 



Two possible options of two-way set partitioning are given in Fig. 9(a) Fig. 9(b). Although the 
10 partitioning of Fig. 9(a) is based on the conventional two-way partitioning of the basic 2D 16- 
QAM constellation (that results from 2 consecutive ID symbols each taken from a 4-PAM 
constellation), it is clear that the one of Fig. 9(b) is better, and if used in the 2D-RSSE for the 
appropriate matrix F0, it will result in better performance, due to the larger "transformed 
intrasubset distance", d l m i n (E'l) in the partitioning. 
15 While in the partitioning of figure 9(a) we get d , m i n A 2(El) = 1.01, which is reached by 

the neighboring points of both subset "a" and subset "b", in figure 9(b) we get d 9 n ^ a ^2(El) - 
1.81, which is also reached by the neighboring points of both subset "a" and subset "b". 
Specifically, for the 2D approach, when the basic input symbols are drawn from a ID PAM 
constellation, it is easy to show that the worst case, in terms of d , m i n (E , l), arises when F0 
20 follows 



1 i + i 
0 1 



in which case, both 2- way set partitioning options achieve d'min A 2(E'l) « 1.25 (assuming a 
large enough ID constellation). 

Taking into account constellation boundary effects, the worst case will be when F0 
25 follows i = -1,0, in which case the number of nearest neighbors in each subset will be 
maximized. 

This case (for F0 following i = 0, and for a 4-PAM basic constellation) can be seen in 
figure 9(c) and 9(d). In figure 9(c) the set partitioning is based on the conventional 16-QAM 
set partitioning, in 9(d) a second possible set partitioning is given achieving the same value of 
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d'min A 2(E'l) for the specific Fq discussed in this example. 

Reference is also made to Fig. 4, which is a schematic illustration of producing an MD- 
subset trellis 38 for an exemplary channel, in accordance with an embodiment of the present 
invention. Producing an MD-subset trellis 38 includes defining the subset states 44 of the MD- 
5 subset trellis and branches 48 of the MD-subset trellis, as is now described. The multi- 
dimensional transmitted (input) symbols x n , which in this example are two-dimensional (2D) 
symbols, takes values from the 4 element set: 

-tl-.M::]) - 

Optionally, the possible values of the multi-dimensional transmitted symbols x n are 
10 partitioned using a 2-way 2-dimensional set partitioning as follows: 

{[!]•[:!]}•{[-:]■[-:]} 

2D-ML trellis states p n ={xn-l> Xn-2) are defined, and 2D-subset trellis states t n ={a n -i(l), 
a n-2( 2 )} are defined, representing the subsets of the two most recent symbols in the respective 
partitioning £2(1) and £2(2). If, for example, Jj - J 2 = 2, and £2(1) and £2(2) follow the 2-way 

15 set partitioning given above, the 4 subset states of the 2D-subset trellis are generated (58) from 
all possible permutations of 2D-symbol subsets. For the example of Fig. 3, four 2D-subset 
trellis subset-states 44 are created. 

Each state 44 represents the possibility that, at time n, the L last multi-dimensional 
symbols {x n -l> Xn-L,} had values included in the subsets of the specific state. For 

20 illustration purposes, the states 44 are duplicated twice, once for the current time n (states 44) 
and once for a following time n+1 (next states 46). From each state 44, J] distinct branches 48 
representing the symbol groups to which transmitted MD-symbol x n m *y belong, lead from 
current subset-states 44 to next subset-states 46. Branches 48 are created (60) by determining 
for each subset-state 48 and each currently transmitted multi-dimensional symbol x n , the next 

25 state 46 to which the sequence x n , Xn- 1 , . . . , *n-L+ 1 belongs. 

In some embodiments of the invention, the created trellis is searched (62) for a given 
received sequence of symbols, so as to provide a sequence of detected input (transmitted) 
symbols. One exemplary method for searching the trellis is described hereinbelow with 
reference to Fig. 6. 
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In some embodiments of the invention, the method of Fig. 2 for generating an MD- 
subset trellis is performed by a manufacturer of an equalizer in manufacturing and/or 
calibrating the equalizer. Alternatively or additionally, one or more of the acts of the method of 
Fig. 2 are performed by the equalizer periodically and/or each time a decoding is to be 
5 performed. This alternative may be used, for example, when the characteristics of channel 24 
change substantially within short periods, e.g., when receiver 26 comprises a cellular receiver. 
The method of Fig. 2 may be performed, for example, each time the receiver is turned on 
and/or when the receiver moves between base stations. 

In some embodiments of the invention, using the method of Fig. 2 or any other method, 

10 an MD-subset trellis is formed in which the movement between a current subset state and a 
next subset state is based on a plurality of consecutively transmitted MD-symbols and at least 
some of the states of the MD-subset trellis represent a plurality of MD symbol sequences. The 
use of such a trellis provides in many cases a substantially lower decoding error rate relative to 
other trellises, for a given computational complexity level. 

15 Fig. 5 is a schematic block diagram of a data structure 70 managed by an equalizer, in 

accordance with an embodiment of the present invention. Data structure 70 includes a 
description 72 of a MD-subset trellis which is used by the equalizer. Trellis description 72 
includes a listing 74 of the MD-subsets included in each of the subset states of the MD-subset 
trellis for each subset state or the MD-ML states (p^) included in the MD-subset states (y. In 

20 addition, trellis description 72 includes a branch listing 76 which lists for each subset-state of 
the MD-subset trellis, the subset-states to which branches from the subset-state lead in a 
following time (n+1). Branch listing 76 also lists for each branch the corresponding MD- 
symbol or subset. Alternatively, the corresponding MD symbol or subset is listed separately. 
Alternatively or additionally, branch listing 76 lists for each subset-state, the subset-states from 

25 which the subset-state has leading branches in a previous time. 

In addition, data structure 70 includes a register array 78 which includes a register 80 
for each subset-state of the trellis, for a plurality of times 5. Register 80 comprises a sequence 
field 82 which stores a most likely symbol sequence (i.e., a path history) from the group of 
symbol sequences of the respective subset-state, a cost field 84 which represents the chances 

30 that the symbol sequence listed in sequence field 82 was the sequence transmitted by 
transmitter 22, given the signals r n received by receiver 26 and a previous subset-state field 86 
which indicates the previous time subset-state from which the current subset-state was 
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reached. Optionally, the cost field is used only for the current subset-states. Optionally, field 
86 is not used, and decisions are made according to sequence field 82. Optionally, for states 
which represent only a single symbol sequence, sequence field 82 is not used. Alternatively, 
for states which represent only a single symbol sequence, register 80 does not include sequence 
5 field 82. 

In some embodiments of the invention, data structure 70 also includes a function 
description 88 describing the model of channel 24. 

Fig. 6 is a flowchart of the acts performed by an equalizer in detecting received signals, 
in accordance with an embodiment of the present invention. The equalizer receives (100) a 
10 sequence of M received symbols r„={r n +M-l> r n }, M being the number of symbols 

required to perform a transition in the trellis of description 72, i.e., the dimensionality of the 
MD model. 

If (102) all the groups of multi-dimensional symbol x n represent only a single MD- 
symbol, i.e., Ji=D M , the equalizer computes (104) for each of the branches of the trellis, as 
15 listed in branch listing 76, a cost of passing on that branch, given the channel function 
description 88 and the received MD-sample r_ n . Optionally, the cost of passing the branch is 

defined as the Euclidean distance between the received MD-sample r_ n and the calculated 

result of passing the transmitted symbols represented by the MD-subset state from which the 
branch leaves and the MD-symbol represented by the branch through channel 24 as modeled 
20 by function description 88 (Fig. 5). In some embodiments of the invention, the branch costs are 
calculated using the following equation: 

in which B T ji is the cost of the transition leading from state i to state 1, x n is the MD- 

input-symbol corresponding to the branch, P n (t n ) represent the L most recent M-dimensional 
25 symbols stored in the path history associated with the subset-state t n (field 82 of the register 80 
of state i), Fq is the first matrix of the MD representation of the channel 24, F = [Fj, F2, ... > 
Fl] are the L matrices representing the channel 24, and "( , ) M denotes the inner product (of the 
subset-state (using the path history) and the L channel matrices) defined by: 
L 

*=i 
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The symbol || || denotes the Euclidean distance. Alternatively, any other distance metric is used, 
such as any simplified approximation of the Euclidean distance. 

If (102), however, some of the subsets of multi-dimensional symbol x n _, represent a 
plurality of MD-symbols, parallel transitions may exist in the MD-subset-trellis (i.e. transitions 

5 that start from a common state and end in a common state may represent different MD- 
symbols x n ). If, for example, all the subsets include the same number of MD-symbols, each 
distinct branch represents D M /Ji parallel transitions. Optionally in these cases, the equalizer 
computes the cost of each branch by finding the minimal cost of each group of parallel 
transitions. In some embodiments of the invention, the equalizer computes (106) the cost of 

10 the branch for each MD-symbol in the parallel transitions (MD-subset) represented by the 
branch, and chooses (108) the lowest cost for passing the branch. In addition to storing the 
lowest cost (in field 84 (Fig. 5), the MD-symbol which resulted in the lowest score is stored in 
field 82 (Fig. 5). In some embodiments of the invention, the branch metric is calculated 
according to : 

= * SRn lir " " (f)M - F) ~ F ^ 2 

15 

where the minimum is computed over all MD-symbols x n in each subset Rj(l) of Q(l) for (1 
<= j <= Ji), corresponding to the transition xj j. 

Alternatively, the MD-symbol which results in the lowest score is selected using slicing 
methods, without explicitly calculating each transition cost. The transition cost is then 

20 calculated only for the transition with the lowest cost for each group of parallel transitions. 

Using the calculated branch costs, the equalizer calculates (110) new state costs for 
each of the states 44 of the trellis. For every state, the path with the smallest metric that 
reaches that state is found. The new state metric is a sum of the previous state's metric and the 
metric of the branch leading from the previous to the current state. For each state, the equalizer 

25 stores in registers 80 of time n, the new state cost (in field 84), the MD-symbol of the selected 
transition (in field 82) and the previous state from which the branch entering the state came (in 
field 86). In mathematical notation, the state's new metric is calculated according to; 

Vjn) = njjg{7*(n - 1) H- } VleM (49) 
where Pj(n) is the metric of state 1 at the n'th instant. M\ is the set of all states that have 
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branches leading to state 1, and M is the set of all states. 

In some embodiments of the invention, the equalizer then finds (112) the state with the 
lowest cost and traverses the trellis backwards (using the best path history of the selected state 
stored in fields 82) for a predetermined number of steps (8), and the MD-symbol stored in the 
5 backward traversed field 82, e.g., x n _ St is provided (114) at the output of the equalizer. In 
some embodiments of the invention, the method of Fig. 6 is repeated a plurality of times for an 
entire received symbol sequence. In an exemplary embodiment of the invention (5) is between 
2L and 4L. 

Alternatively to finding (1 12) the lowest cost state and providing an output MD-symbol 

10 for each trellis iteration performed, a plurality of symbols are provided every predetermined 
number of trellis iterations. In this alternative, the equalizer finds the state with the lowest cost 
every predetermined number of iterations, traverses back from the state with the lowest cost 8 
steps and then provides as output a sequence of MD-symbols. Optionally the equalizer may 
use some other decision rule to determine the state from which to begin to traverse backwards. 

15 In some embodiments of the invention, the provided MD-symbol (or symbols) are 

converted to one dimensional symbols before they are provided by the equalizer. 

Alternatively to computing (106), for each transition, the cost of passing the branch for 
each MD-symbol in each subset represented by the branch and choosing (108) to represent the 
branch the MD-symbol resulting with the lowest cost, the equalizer determines the MD- 

20 symbol which will have the lowest cost using a slicing method known in the art and calculates 
the cost of the branch only for the determined sequence. 

As described above, the specific MD-RSSE trellis used may be chosen responsive to 
the computation and/or space complexity of using the specific trellis. In some embodiments of 
the invention, the specific MD-RSSE used is chosen based on the performance characteristics 

25 of the MD-RSSE. In some embodiments of the invention, the performance is estimated using 
simulations. Alternatively or additionally, the complexity of the specific MD-RSSE trellis is 
estimated as is know described. 

In some embodiments of the invention, the space complexity required for performing 
the detection is estimated as being proportional to the number of states in the trellis. For the 

30 MD-Subset trellis, the number of states is: 

A r =IH (CD) 
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and the space complexity of the MD-RSSE is o(N). 

The computational complexity of the MD-RSSE can optionally be estimated as being 
of the order of the number of states in the trellis, multiplied by the number of branches leaving 
each state. 

5 In an alternative embodiment of the invention, the computational complexity is 

represented by one or more of the following three different categories, which include the 
number of branch metric calculations, the number of summations needed for path metric 
computation, and the number of comparisons. Without loss of generality, in this discussion, for 
simplicity reasons, we shall consider symmetric set partitioning, such that all subsets Rj(l) (1 
10 <= j <= Jl) in the partitioning £1(1) consist of the same number of elements. The number of 
states in the MD-S trellis follows N, whereas the number of branches to and from each state is 
D A M. 

When J i < EKM these branches may be divided into Jj groups of parallel transitions, 
each consisting of D A M/Jj branches (in the symmetric case). 
15 As described above, in some embodiment of the invention, each iteration of the method 

of Fig. 6 used to search the MD-S trellis includes: 

(a) If Jl = D A M use step (al), if J_l < D A M use step (a2). (al) D A M branch metrics are 
calculated for each state, leading to N*D A M calculations. (a2) Using brute force methods, 
D A M branch metrics are calculated for each state, and delay-free decisions are made between 
20 every D A M/J1 symbols, for each subset in the partitioning £2(1), leading to N*D A M 
calculations, and N*J1(D A M/JM) comparisons. (b)N*Jl path metrics are calculated, as the 
sum of previous path metrics and branch metrics, (c) A minimal path metric is found out of Jl 
metrics, for each of the N states, leading to N(J1-1) comparisons, (d) One minimal state is 
chosen out of N states, leading to N-l comparisons. 
25 In some embodiments of the invention, when Jl = D A M, and when Jl < D A M the total 

number of operations is as follows: 

Number of branch metric calculations: N*D A M 
Number of summations: N*J1 
Number of comparisons: N*D A M - 1 
30 Notice the linear dependency of the total number of operations on the number of states 

in the trellis N. 

In some embodiments of the invention it is desirable to choose Jj as small as possible, 
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without causing d'min to become too small (through early merging error events of length 1) 
which may lead to a degradation in performance. Also notice the exponential dependency on 
the size of M, the dimensionality of the multi-dimensional approach used. 

On the other hand, each iteration of the MD-RSSE results in estimating an M- 
5 dimensional symbol (i.e. optionally M consecutive symbols), and thus the number of 
operations at each iteration should be divided by M, when compared with classical, one- 
dimensional methods. 

In some embodiments of the invention, the branch metric calculation is performed by 
finding the minimum squared Euclidean distance between pairs of points in the M-dimensional 
10 space. Optionally other types of distances may be used. A first point in the pair is defined as: 

i = i,-fc(t n ].r) (ci) 

Point z' n is associated with the received multi-dimensional signal r n , and the path history of 
the subset state t n . A second point in the pair is defined as: 

^ = Fo£, (02) 

15 which is a linear transformation of all MD-constellation points in the respective multi- 
dimensional subset of £2(1). Optionally, the form of Fq is used to derive an efficient way of 
finding the closest multi-dimensional subset point, even if it is difficult to use slicing in order 
to find the closest multi-dimensional symbol in the linearly transformed subset. Particularly, 
this method may be used when the partitioning Q(l^has a symmetry. Optionally, when Fq is 

20 of the form: 



Jo h /u-2 /.U-l 
0 Jq . - , /u-n Af-2 

0 0 ... 0 /□ 



(68) 



the closest multi-dimensional transformed subset point is found in an iterative way. The M'th 
dimension is scaled by a factor fg , and a decision is made in the M ! th dimension using a 
slicing operation. 

25 Denoting the i'th element of a vector y as v (,) , once the M'th element of x n > namely 

x n (M) is known, it is used to find the (M-l) f th element, which is transformed to be: 
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using a simple slicing operation. In some embodiments of the invention, this procedure is 
repeated for all M dimensions, yielding for each state a total of only J} explicit branch metric 
computations, one for each multi-dimensional subset of the partitioning £2(1). Optionally, the 
5 temporary results, i.e. the branch metrics of each dimensions, used for the iterative slicing 
procedure, can be used for the computation of the branch metric, without re-computing them. 

Alternatively, any other method, such as described in L.F. Wei, "Trellis-coded 
modulation with multi-dimensional constellation", IEEE Trans. Inform. Theory, Vol. 33, No. 
4, July 1987, is used to determine the symbol sequence with the lowest cost without 

10 calculating the cost for each of the symbol sequences of the branch. Optionally, the 
partitioning of a multi-dimensional lattice is performed in an iterative manner. That is, the 
partitioning of a 2M-dimensional lattice is based on the partitioning of the constituent M- 
dimensional lattices, which is in turn based on the partitioning of the constituent M/2- 
dimensional lattices. Based on dividing the basic (optionally ID or 2D) constellation into 

15 subsets, multi-dimensional types are constructed by concatenating basic subsets, while multi- 
dimensional subsets, sub-families and families are constructed by grouping together several 
multi-dimensional types, subsets and sub-families respectively. Following this method of 
constructing the multi-dimensional partitioning, a simplified maximum-likelihood decoding 
algorithm is given, in which the point in each multi-dimensional subset closest to the received 

20 multi-dimensional point is also found in an iterative way. The multi-dimensional received 
signal is divided into several constituent signals lying in a space with the basic dimensionality 
(say M). Each received 2M-dimensional point is divided into a pair of M-dimensional points. 
The closest point in each 2M-dimensional subset and its associated metric is found based on 
the point in each of the M-dimensional subsets which is closest to the corresponding received 

25 M-dimensional point and its associated metric. The M-dimensional subsets, are optionally 
those that were used to construct the 2M-dimensional subsets. This process is optionally 
performed iteratively to obtain the closest point in each 2M-dimensional subset and its 
associated metric, based on the closest point in each of the basic (optionally ID or 2D) subsets 
and its associated metric. 

30 Alternatively or additionally, when the multi-dimensional partitioning £2(1) used for 

generating the MD-subset trellis is similar to the MD-set partitioning described in L.F. Wei, 



19 



WO 01/28174 



PCT/US0O/28536 



"Trellis-coded modulation with multi-dimensional constellation", IEEE Trans. Inform. Theory, 
Vol. 33, No. 4, July 1987, iterative methods are used as described above, in order to reduce the 
number of calculations needed for each iteration of the method of Fig. 6. Alternatively, a 
combination of some or all of the methods described above is used. 
5 Referring in more detail to determining a probability of error parameter for use in 

selecting a partitioning to subsets, following is an optional procedure for calculating a 
parameter of a probability of symbol error. 

An error event E is said to occur , extending from time k\ to lq+i, if the estimated state 

sequence {Pn(^ri)} is equal t0 ^ correct state sequence {P n (t n )} at times k l and k l+I but 
10 nowhere in between. For that we define the error sequence e , corresponding to £ as: 

e = fefc^fifci+n-- -lfiifei-H-i] 

where each element of e follows: 

Each error sequence e corresponding to the error event E obeys the three rules: (a) The element 

15 e^i is nonzero, (b) The last L elements, eki+i-L through eki+l-i satisfy the "subset merging 
condition", and so do the L elements prior to the beginning of e. This means that eki+i_L.i+k 
is equal to fekl+l-L-l+k ~ ^lkl+1-L-l+k ) where the correct multi-dimensional input symbol 
£kl+l-L-l+k and estimated symbol xlkl+l-L-l+k belong to the same subset in the 
partitioning £2(k) for k = 1,2,...,L. (c) No L elements of e prior to the last ones satisfy the 

20 "subset merging condition". 

When MD-ML states are used, two paths merge only if their last L elements are the 
same. When MD subset states are used, paths merge in the MD-Subset trellis if the subsets of 
their last L elements are the same. This difference can be seen in rule (b) above. Moreover, 
generally an error event in the MD-RSSE (i.e., the method in which the MD-subset trellis is 

25 used) may occur, even if the estimated subset state sequence {t' n } is equal to the correct state 
sequence {t n }. When Jfc = D M for all k, there is no difference between the MD-RSSE, and the 
MD-MLSE (the method in which the MD-ML trellis is used), but generally the MD-S (MD- 
subset) trellis will have a set of what may be called "early merging error events" which are not 
present in the MD-ML trellis. On the other hand, some error events in the MD-ML trellis, may 

30 not belong to the set of error events of the MD-S trellis. This may occur for instance for MD- 
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ML trellis error events for which L elements, prior to the last ones, satisfy rule (c) above, 
which holds a weaker condition than the corresponding one for MD-MLSE. 

When J i < D M , parallel transitions appear in the MD-S trellis. In this case, error events 
of length 1, associated with the parallel transitions are part of the "early merging error events". 
Another difference between MD-RSSE and MD-MLSE concerns the phenomenon of error 
propagation. In the MD-RSSE, when the correct path is discarded in the subset trellis, it is 
possible, due to the use of decision feedback, to have error propagation. It is possible to 
analyze the effect of error propagation analytically (e.g. using state machine models), or assess 
the effect of error propagation through simulations. In the analysis given here the probability of 
a first error event occurring at some time k\ is analytically analyzed. This gives close 
approximation to the symbol error probability. 

Following the definition of the error sequence e related to the input symbols, an error 
sequence e y associated with the signal sequence {y n } is defined as: 



-j 



where each element of ey follows: 
~ def 

The squared distance d 2 (e) of an error event e is defined as: 

<<■<«) Elk" 

k=ki 
ki-H 

= E 

ki-H 

= E 

k=ki 



3=0 



Foe* + Y.Fj^.j 



where ^=0 for kj-L <= k <= kj-1, following condition (b) above, and the fact that we deal 
with a first error event. The number of errors in the error sequence e associated with the error 
event E is also defined. w(e) is defined, for example, as the Hamming weight of the error event 
E, with which the error sequence e is associated. 
For the first error event E to occur, three sub-events typically occur: 
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• At timeAi {p kl (t kl )} = 

• £ 7 : Between k x and k x + Z- 1 the input ^TObols^a*,^- - must be such 
that when added to the error sequence e. they result in an allowable sequence. 

• £3: For k% <k<ki+L the sequence must haw greater likelihood than {x^}. 
For E3 to occur we get: 

As mentioned herein, E3 can occur without actually leaving the correct subset state sequence 
5 {t n } in the MD-S trellis. 

The probability of the sub-event E2 depends only on the statistical properties of the 
input sequence, and on the value of the error sequence e. Furthermore it is independent of E] 
and E3. Let us consider a PAM (pulse amplitude modulation) case known in the art where the 
input sequence is independent and equiprobable, taken from an alphabet of size D (assumed 
10 even) {-(D-l), -1, +1, +3, .,.,D-1 },then: 

p'<^}= n (so) 

Optionally, input sequences other than PAM can be dealt with in a similar manner. The 
probability that E\ is not true, is of the order of the error probability, and therefore: 

Pr{&|5 3 } 

15 is well approximated as well as upper bounded by 1, for reasonably low symbol error 
probabilities. Altogether the probability of a particular error event E, to which the error 
sequence e is associated, is given by: 

Let F be the set of all first error events E, starting at time ki . Then using the union bound, the 
20 probability that a first error event starts at time kj is upper bounded by 

FMtf}<£Pr{£} (52) 
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let us separate the set E 1 into several distinct sets: (a) The set E'mL of a11 error events that 
belong to the set E of all error events E, starting at time k\ in the MD-ML trellis, (b) The set 
E' i2 of all the "early merging error events" E, starting at time kj, which are not present in the 
MD-ML trellis. When Jj < D M , the set E 9 \2 may be represented by the disjoint union of E* \2 

5 = E'i U E*2 where: (bl) The set E'] represents all early merging error events of length 1, 
associated with parallel transitions in the MD-S trellis, at time k\. (bl) The set E'2 represents 
all other early merging error events in the MD-S trellis, starting at time k[. 

Letting G be the set of all d(e) , and for each d(e) belonging to G letting E'<j be the 
subset of all error events for which d(e) = d, the probability of a symbol error can be upper 

10 bounded by: 



20 



n u^ 2 ^- 



(53) 



Due to the exponential nature of the Q-function, the expression is dominated at moderate SNR 
by the term involving the minimum value d'^^m of d(e) , for all error sequences e that 
correspond to the error events E belonging to E\ 
15 d'min can be written as: 

drmn = niin( niin rffel. minc/ie). ininc/Cel) 

= mBatdnfc niin 4 el. niin<#e)l * } 

Let us now consider the early merging error events of length 1. The minimum distance 
between all parallel transitions should be calculated according to all the possible multi- 
dimensional intra-subset errors e^for the partitioning fl(l) so that 



Altogether: 

dndn = ™n(d«£ n , tt™i»{&^. d^E'i)) (50) 

where d i n ^ n (E* 1 ) is given above and d*^^ (E'2 ) is the minimum distance among the 
remaining early merging error events that are longer than one multi-dimensional symbol 
25 interval. Therefore care should be taken as to the partitioning Q(l) and to the value of Jl , 
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taking into consideration also the value of Fq . Now, based on the probability of first error 
events (i.e. neglecting the effect of error propagation) the symbol error probability can be 
approximated by 



(57) 



5 where for the case of PAM, as discussed above 



n n ] kA 



2D 



(58) 



Therefore, for approximating the symbol error probability for a given channel and a given 
transmission scheme and noise variance, we may determine the value of d' n ^ n , and then use 
10 Kd'min- Optionally the noise variance (sigma) may also be approximated, if not known, by 
some method known to the art. Alternatively, for the case of QAM (Quadrature amplitude 
modulation) transmission, the same method may be performed, only Kd'min ls found to be: 



20 



A'«w= £ We) 



>g-> || f 2D-\Re{e k j}\ \ / 2£)-|/m{e fcj }| V 
I k=ki j=l V 2 ^ / V 2£) /. 



In some embodiments of the invention, Pr{e} is used as a criteria parameter in 
1 5 selecting a subset partitioning in the method of Fig. 2. 

Following is an example of using the MD-RSSE, for a binary transmission through a 
finite impulse response (FIR) filter having the following characteristic: 

/o = l 

/ = [/i,/ 2 ,/ 3 ,/i] = [0.1,-0.1,0.2,0.8] 

Leading to the following input-output relation: 
A 

= x n + QAx n -i - 0.Lr„_ 2 + 0Jx n ^ + 0.83^ -f w n 



The energy of the filter is: 
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such that in the output sequence, only 59% of the energy of the input symbols will be 
embedded in the cursor term, while 41% will be embedded in ISI terms. The severity of the 
channel can also be seen from its frequency response having attenuation of over 20dB. 
Examination of the dn^ property of this channel, with a binary input, yields the following; 

<Ln = 6.8 

corresponding to the error sequence of length 5 having the form: 

e m t„ = [±2,0,0,0,0] 

In a 2 -dimensional approach: 





Z3 






[:]- 


* 2 


5 ai2 = 
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1 




1 




-1 




1 




-1 




1 





:!]} *«■ 



£o = 



r 0 



:Ll = 



r 3 

*2 



-.La = 



The three 2*2 matrices (following L = 2) describing the channel, are: 



F 0 = 



1 0.1 
0 1 



" -0.1 


0.2 ' 




' 0.8 


0 " 


0.1 


-0.1 


,F 2 = 


0.2 


°- 8 



1 5 and the relation between the input and output is modeled by: 
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ft=0 





' 1 


0.1 ' 






0 


1 





' -0.1 


0.2 " 




' 0.8 


0 " 


0.1 


-0.1 




0.2 


0.8 



In the MLSE model, trellis states are optionally defined as: 

Since each 2-dimensional element of the state vector can take one of 4 values, the 2D-ML 
trellis has 16 states, with 4 transitions to and from each state. In some embodiments of the 
invention, the Viterbi algorithm (VA) is used to search the 2D-ML trellis as described above. 

In some embodiments of the invention, a 2- way 2-dimensional set partitioning in which 
J], J2 = 2 is used and Q(\) and Q(2) are chosen as: 



{[!]■[::]}•{ 



!]} 



10 The subset trellis of a 2-dimensional sequence at time n is defined as: t n =[a n _i(l), a n _2(2)] 
representing the subsets of the two most recent symbols in the respective partitioning Q(l) and 
£}(2). Since each element of t n can take on one of two values, the total number of states in the 
subset trellis is 4, with J} = 2 transitions to and from each state. It is noted that each branch 
represents 2 parallel transitions that end in the same state. 

15 The possible 2-dimensional intra-subset errors for the partitioning Q(l) are: 





2 ' 




2 " 


£11 =± 


2 


, £12 = ± 


-2 



which lead to error events of length 1, associated with parallel transitions having the following 
values of d' 2 : 



df 2 (e l2 ) = 7.24 > <C 
20 Thus: 
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cf mm (EU) = 7.24 > 4, n 



and the 2D-RSSE performance shall not be dominated the parallel transitions in the search of 
the subset trellis. A close examination of the subset trellis reveals that 

which is achieved by the 2-dimensional error events: 



-nun 3. 



-i = ( ± 



2 




0 




0 


0 




0 




0 


0 




0 




0 


2 




0 




0 



) 
) 



meaning that in this particular example no severe degradation is expected in comparison to the 
full 16-state MLSE due to early merging error events. Some degradation is expected, however, 
in this particular example due to the larger number of error events having the d m j n , i.e., 

10 Kd'min ~ 2 compared to Kjmin = 1 for the case of the full 16-state MLSE. 

The above described scheme performs very closely to the full MLSE (having 16 states), 
and is superior over the prior art RSSE/DFSE methods by 2 dB at moderate to high SNR, even 
though it has less complexity. The above analysis shows that for the case discussed here, the 
method of the present invention outperforms tremendously the prior art methods, in terms of 

1 5 performance/complexity tradeoff. 

Following is another example of using the MD-RSSE, for a transmission using a 16- 
QAM constellation through a finite impulse response (FIR) filter having the following 
characteristic: 

/o = 3 

/ = l/i* h h, U. /v / 6? Si, Ss\ = [Q05. 0.10. 0.20. 0.05. 0.30. 0.10. 0.20. 0.73] 

20 Leading to the following input-output relation: 
a 

= At + 0.05^- 1 - 0. Lr.,-2 + 0.2^^,3 + Om^i + 0.3^-r, + G 1 a-q 
+0.2:r n _7 + 0. Tax*-* + "<* 
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10 



The energy of the filter, is: 
l/l 3 = E A 2 =1.3373 

such that in the output sequence, only 57% of the energy of the input symbols will be 
embedded in the cursor term, while 43% will be embedded in ISI terms. The severity of the 
channel can also be seen from its frequency response having attenuation of almost 40 dB. For 
the case discussed here, the full MLSE has high complexity due to the large signal 
constellation and the long impulse response of the channel. It is interesting to analyze it, in 
order to achieve theoretical lower bounds on the expected performance of any scheme. 

An examination of the d m ^ n property of this channel, with a 16-QAM constellation, 
yields the following: 

<4« = 1-758 

corresponding to the four error sequences of length 9 having the form: 
c™, = [=1,0,0,0,0.0,0,0.0] 

^2 = 1^,0,0,0,0,0,0,0,0] 
Let us now adopt a 2-dimensional approach, leading to the following definitions: 



15 



20 



20 = 



Xi 

X 0 



■£l = 



*3 
*1 



:X2 = 



is the 2-dimensional input sequence. Since the basic input symbols where taken from a 2- 
dimensional 16-QAM constellation, this scheme may be interpreted as 4-dimensional. 

The input 2-eIement vectors are taken from the 4-dimensional constellation having 256 
possible values. Each constituent 2-dimensional element of x„ is taken from the 16-QAM 
constellation of figure 8. The 2-dimensional output sequence is: 



n 




r 3 




" T 3 




:£ 2 = 




r-2 



where again, each element is taken from a 2-dimensional plain, and 
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F 0 = 



1 
0 

0.1 
0.3 



0.05 
1 

0.2 
0.1 



■ F 4 = 



0.1 0.2 
0.05 0.1 

' 0.75 0 
0.2 0.75 



0.05 
0.2 



0.3 

0.05 



are the five 2*2 matrices (following L = 4) describing the channel. 

Using the above definitions, a simple relation between input and output may be defined as: 



4 

I 

*=0 





' 1 


0.05 ' 




0.1 


0.2 ' 






0 


1 _ 




0.05 


0.1 





0.75 0 
0.2 0.75 



0.05 0.3 
0.2 0.05 



2^-2 + 



0.1 0.2] 
0.3 0.1 J 



5 where w„ represents a 2-dimensionaI complex noise sequence. Let us first view the 
construction of the 2-dimensional MLSE. 
The trellis states shall be defined as: 

since each 2-dimensional element of the state vector can take one of 256 values, the 2D-ML 
10 trellis has 256 A 4 states, with 256 transitions to and from each state, and is enormously 

complex to implement. Theoretically, a VA can be used to search the 2D-ML. 

Optionally a 2-way 4-dimensional set partitioning, based on Wei, known in the art may 

be used. The partitioning is based on the partitioning of the 2-dimensional 16-QAM 

constellation into 4 subsets as shown in Fig. 8, which is identical to the 4-way Ungerboeck set 
15 partitioning of the 16-QAM constellation. Sixteen 4-dimensional types are created using the 

concatenation of two 2-dimensional subsets, as given in the following table: 
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'ID family 


4D subset 


ID types 


1 


1 


(a,a) 






(b ; b) 




2 


(c 5 c) 






(d,d) 




3 


(a,b) 






(b.a) 




A 


M) 






(d,c) 


2 


5 


(a,c) 






(M) 




6 


(c,b) 






(cU) 




7 


(a 5 d) 






(b 5 c) 




8 


<<*0 






(d,b) 



Eight 4-dimensional subsets are generated by grouping each two 4-dimensional types together, 
keeping a squared intrasubset distance of: 

A* = 4 

Next, two 4-dimensional families are generated by grouping four 4-dimensional subsets 
together. The squared intra-family distance is reduced to: 
A} = 2 

The two 4-dimensional families are used in order to generate the MD-RSSE. Notice that this 
4-dimensional partitioning may be used for any basic rectangular 2-dimensional constellation. 
Letting Jl = J2 - J3 = J4 = 2, so Omega(l) through Omega(4) follow the 4-dimensional two- 
family partitioning of the table. This allows us to define the subset trellis of a 2-dimensional 
sequence at time n as: 

t n = [a n . 1 (l),G fl - 2 (2),a n . 3 (3),a Tt - 4 (4)] 

representing the subsets of the four most recent symbols in the respective partitioning 
Omega(l) through Omega(4). Since each element of t„ can take one of two values, the total 
number of states in the subset trellis is 16, with Jl = 2 distinct transitions to and from each 
state. Notice also the existence of 128 parallel transitions that start in a common state and end 
in a common state. 
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Using the VA to search the subset trellis we get the 16-state 2-dimensional RSSE (2D- 
RSSE). Going over all possible 2-dimensional intrasubset errors el for the partitioning 
Omega(l), reveals that: 

<^0 = 1-9025 
where the minimum is attained for 





" -j ' 




" -1 " 




j . 


) i-;2 


1 



10 



so the 2D-RSSE performance shall not be dominated by the parallel transitions (i.e. error 
events of length 1) in the search of the subset trellis. 
A close examination of the subset trellis reveals that 



dliin = = 1-758 
which is achieved by the 2-dimensional error events: 
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0 




0 
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0 




0 
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' 0 " 




" 0 " 




' 0 " 




' 0 " 
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> 
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meaning that no severe degradation is expected in comparison to the full 16 A 8-state MLSE due 
to early merging error events. Still some degradation is expected due to the larger number of 
15 error events having the minimum distance dmin, resulting in Kdmin = 6, instead of Kdmin = 3 
for the full MLSE. 

Analysis and simulations show that the new scheme performs very closely to the full 
MLSE (having 16 A 8 states), and is superior to all other practical schemes by more than 2 dB at 
moderate to high SNR. While the space complexity of the 2D-RSSE is very low, due to the 16- 
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state 2D-S trellis in use, it is of interest to estimate the computational complexity of the new 
scheme. The computational complexity in this case, might seem at first glance intimidating, 
due to the large number of branches leaving and entering each state (and hence branch metrics 
to be calculated). The number of branches here, following D = 16 and M ~ 2, is 16 A 2 » 256. 

5 Notice that these branches are divided into J_l = 2 groups, each consisting of 256/2 = 128 
branches (or parallel transitions). A brute force method, may require at each iteration and for 
every state of the trellis, the computation of up to 256 branch metrics, followed by selecting 
the smallest metric from each group of 128 parallel transitions. 

In order to reduce the number of calculations needed at each iteration, some 

10 embodiments of the invention make use of the methods to reduce complexity of each iteration, 
with adaptations to this special case. In some embodiments of the invention, the following 
algorithm, which is based on the decomposition of the branch metric computation is used; 

= in nnn o {|r<0 - (p n (t n )..Fp - |/ 0 4 2) ff 

+" ||r^ - (p n (t n ), F)& - [0 /oW 4Tf } 
= ^ { ||r<0 - (p n ( tn ), Fp - /erf) - /.4i 2 



where 



15 





r i 















and 





I I 




' 1 0.05 " 


0 






0 1 



Rj(l) for j = 1,2 are the two 4D families of the table, and ( )T stands for the transpose 
operation. 

20 This leads to the following algorithm: 
(1) For every state in the trellis, compute 



32 



WO 01/28174 



PCT/US00/28536 



r<?>-(p n (t n ).F)W. 

and find the closest point X (2) n in each 2D subset (i.e. subsets {a, b, c, d}, following figure 8) 
and its metric. This may be done using simple slicing operations. Altogether it involves 4 
slicing operations and 4 2D branch metric computations. 
5 (2) For the selected point in each 2D subset, X* 2) n , compute 

and find the closest point X (1) n in each union of 2D subset pair (i.e. subsets {a U b} and {c U 
d}, following figure 8) and its metric. This follows the 2 4D families, as described in the table. 
Notice that when X^n belongs to the set {a U b} than the closest point X (l) n belonging to the 
10 set {a U b} corresponds to the 4D-family 1 in the table, whereas when X* 2 ^ belongs to the set 
{c U d} than the closest point X 0 ^ in {a U b} corresponds to the 4D-family 2. Altogether, this 
step involves 4*2 « 8 slicing operations, and 8 2D branch metric computations. 

(3) Compute the 8 branch metrics of the 4D points corresponding to family 1 (i.e. 
corresponding to the 4D types (a,aUb), (b,aUb), (c,cUd),(d,cUd)) and to family 2 (i.e. 

15 corresponding to the 4D types (a,cUd), (b,cUd), (c,aUb), (d,aUb)). This is done simply by 
adding the results attained from steps 1,2. 

(4) Find the minimum branch metric, and the corresponding 4D symbol Xn , for each 4D- 
family, out of each 4 branch metrics computed in the previous step. This involves 2*3 = 6 
comparisons. In a particular embodiment the proposed algorithm reduces the number of 

20 required operations, for each iteration, and for every state of the trellis to: 
12 slicing operations 

12 computations of 2D branch metrics (plus 8 summations needed for the computation of the 
4D branch metrics) 
6 comparisons. 

25 This is generally better than the brute force method, which may require 256 4D branch metric 
computations and 254 comparisons. Therefore our 2D-RSSE is not only superior to other 
methods in performance, but also feasible to implement. In this example again the new scheme 
outperforms the existing ones in terms of performance/complexity tradeoff. 

Alternatively to forming multi-dimensional symbols from consecutive transmitted 

30 symbols, in some embodiments of the invention, multi-dimensional input symbols are 
combined from M\ consecutive or non consecutive input symbols, and multi-dimensional 
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output samples are combined from M2 consecutive or non consecutive input symbols. 
Optionally M\ may be different than M2 . In That case, the following definitions may be used 
to get a MD-system model: 



2 n 



2Vi 



r nA/ 2 +fto 



F^ 0 where k = {X,(n). Lj(n) + l.L,(n) + 2. .L 2 (n) - 1, L 2 (n)} am L 2 ( n) - Li(n) H- 1 
matrices of size M\ * A/2 (having M\ rows and A/ 2 columns) each, such if tat: 



fn<Sh-*\f\)*k\fir*bra\tr-i ^(A/^Ui^Wn^naw,-? j£i£fe-*/i>*A/rt*i-<ii /n^V/a-A /i>4* A/HA r<io 



where Lj(n) and L2(n) are respectively the smallest and largest values of k for which F (n) k 
10 differs from the zero matrix, and 

{a Q , a u ..., a Ml ^} ={0,1,..., M } - 1} (mod M t ) 
and 



{b Q , 6 is - . . , 6 A/2 _i } = {0, 1, - - , M 2 - 1 } (mod M 2 ) 
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Fig. 7 is a schematic block diagram of a multi-input multi-output (MIMO) symbol 
detector, known in the art. In some embodiment of this invention, Multi-Dimensional methods 
described above are used to recover the inputs of a multiple-input-multiple-output (MIMO) 
system, as shown for example in Fig. 7, based on observations that are made on the outputs of 
5 the system. Such MIMO systems may comprise, for example digital communication, speech 
processing, and/or acoustics systems. The MIMO systems are meant herein to include single- 
input-multiple-output (SIMO) systems, such as occur, for example, in co-channel reception. 
In a M*M channel case, the relation between input and output may be defined as follows: 



10 where x n is the M-dimensional input sequence, consisting of elements from each of the M 
input sequences, such that: 



*=0 




*2n 



*2n 



r n is the M-dimensional output sequence, such that: 



r ln 

*2n 



r 3n 



15 Fk where k={0,l,2, L} are L+l matrices of size M*M each, such that: 



fiik f'2lk /MI* 
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w n represents an M-dimensional noise sequence. 
The multi-dimensional signal sequence: 

L 

is the multi-dimensional convolution of the multi-dimensional input sequence { x n } with the 
5 finite impulse response { Ffc }. 

The principles of the present invention may be applied also to systems with different 
numbers of input and output symbols. Furthermore, the principles of the present invention may 
be used with MD-symbols which represent a plurality of time sequences of multi-inputs and/or 
multi-outputs. 

10 In some embodiments of the invention, the above described method may be used for 

combined detection and decoding. In an exemplary embodiment of the invention, a MD 
approach is used for system modeling for the channel or the code sequence or for both, 
followed by reducing the number of states and distinct transitions in the MD-trellis. Optionally 
a MD-subset trellis is generated such that only branches which are legal in accordance with the 

15 coding scheme used by transmitter 22 remain in the trellis. Optionally a decoder and detector 
for MD-trellis coded modulations known in the art and ISI may be constructed using the above 
described embodiments. 

The above described embodiments may be used in substantially any receiver, including 
receivers of data, voice, image and/or any other data. Furthermore, the above described 

20 embodiments may be used with receivers which receive signals from substantially any medium 
including, wireless mediums (e.g., cellular, radio, satellite) and wired mediums (e.g., cables, 
fiber optics, telephone lines), as well as acoustic communication systems. Furthermore, the 
present invention may be used with point to point or point to multi-point systems. 

In an exemplary application, a cellular telephone, for example using an FDMA, 

25 TDMA, GSM or CDMA communication protocol implements the above equalizer methods, 
for example, as hardwired circuitry, such as an ASIC, or using firmware or software 
programmable circuits. Other exemplary receivers which may incorporate the above equalizer 
appartus (e.g. hardware and/or software designed and/or programmed to implement the 
equalizer methods), include, BlueTooth, cellular base stations, Modems, media readers such as 

30 disk drives and optical media, and cable TV systems. 
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In an exemplary embodiment of the invention, a single equalizer can utilize multiple 
equalization methods, for example, one or more of the above described methods and, 
optionally, one or more multiple prior art methods. Optionally, the equalizer selects the 
method based on the noise levels, other noise characteristics, received signal quality and/or 
5 available computational resources. Optionally, the equalizer includes a table that compares the 
computational requirements and/or signal detection qualities of the plurality of equalization 
methods. 

In an exemplary embodiment of the invention, one or more of the above MD-RSSE 
methods is used to select optimal transmission sequences, instead of or in addition to being 

10 used for extracting received signals, for example at cellular base stations, or for wireless 
transmission from cellular telephones. It is noted that the optimal sequence may change, for 
example, as a function of instantaneous interference characteristics. 

In addition, the present invention may be used with communication channels of 
substantially any interference and/or noise patterns. In an exemplary embodiment of the 

15 invention, the methods are used in applications where multiple reflections and transmission 
paths are expected, such as city networks and home networks. It is noted that although the 
system models described above are used in performing some of the acts of the methods of the 
present invention, these methods may be used even when the models or inaccurate or incorrect. 
Specifically, the noise and/or interference of the model may result from the transmitter, the 

20 channel, the receiver and/or any other elements of the system. Furthermore, the noise and/or 
interference may result from other systems or may be generated intentionally. The present 
invention may be used with analog and/or digital systems. The present invention may also be 
used in systems other than data transmission systems, for example data recovery systems, such 
as magnetic recording systems. 

25 It is further noted that the above equations may represent real and/or complex values. 

Likewise, the system and/or elements thereof may be complex or real. 

It will be appreciated that the above described methods may be varied in many ways, 
including, changing the order of steps, and the exact implementation used. It should also be 
appreciated that the above described description of methods and apparatus are to be 

30 interpreted as including apparatus and software for carrying out the methods and methods of 
using the apparatus. 
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The present invention has been described using non-limiting detailed descriptions of 
embodiments thereof that are provided by way of example and are not intended to limit the 
scope of the invention. It should be understood that features and/or steps described with 
respect to one embodiment may be used with other embodiments and that not all embodiments 
5 of the invention have all of the features and/or steps shown in a particular figure or described 
with respect to one of the embodiments. Variations of embodiments described will occur to 
persons of the art. 

It is noted that some of the above described embodiments describe the best mode 
contemplated by the inventors and therefore include structure, acts or details of structures and 

10 acts that may not be essential to the invention and which are described as examples. Structure 
and acts described herein are replaceable by equivalents which perform the same function, 
even if the structure or acts are different, as known in the art. Therefore, the scope of the 
invention is limited only by the elements and limitations as used in the claims. When used in 
the following claims, the terms "comprise", "include", "have" and their conjugates mean 

15 "including but not limited to". 
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CLAIMS 

1. A method of detecting symbols in a signal transfer system, comprising: 
transferring symbols on a channel which may distort the symbols; 

5 representing the transferred symbols by multi-dimensional (MD) symbols, which each 

includes a plurality of transferred symbols; 

providing a trellis which has states which are defined by one or more MD symbols, and 
which has branches which correspond to respective MD symbols; 
receiving a plurality of samples; and 
10 calculating estimate values of one or more transferred symbols based on the received 

samples and the provided trellis. 

2. A method according to claim I, wherein the MD symbols comprise a plurality of 
transferred symbols which take on values from identical value sets. 

15 

3. A method according to claim 1 or claim 2, wherein at least one of the states of the 
provided trellis comprises a subset state which represents a plurality of different MD-symbol 
values. 

20 4. A method according to claim 3, wherein the trellis comprises states which represent 
values of a predetermined number of MD symbols and wherein providing the trellis comprises 
generating the trellis by partitioning the possible values of each of the MD symbols in the 
states of the trellis into one or more subsets and wherein the states of the trellis comprise all 
the permutations of the subsets of the values of the multi-dimensional symbols. 

25 

5. A method according to claim 4, wherein partitioning the possible values of each of the 
MD symbols in the states of the trellis into one or more subsets comprises grouping the 
possible values of different MD symbols of the trellis states into different numbers of subsets. 

30 6. A method according to claim 4, wherein partitioning the possible values of each of the 
MD symbols in the states of the trellis into one or more subsets comprises partitioning the 
possible values of the MD symbols which include earlier transmitted symbols into no more 
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subsets than the possible values of the MD symbols which were transmitted later were 
partitioned into. 

7. A method according to claim 4, wherein partitioning the possible values of each of the 
5 MD symbols in the states of the trellis into one or more subsets comprises partitioning the 

possible values of later transmitted symbols into subsets of which all the members are included 
in a single subset of earlier transmitted symbols. 

8. A method according to any of the preceding claims, comprising generating a model of 
10 the channel and wherein providing the trellis comprises providing a trellis generated 

responsive to the channel model. 

9. A method according to any of the preceding claims, wherein providing the trellis 
comprises providing a predetermined trellis. 

15 

10. A method according to any of claims 1-8, wherein providing the trellis comprises 
providing a trellis determined responsive to the received samples. 

11. A method according to any of claims 1-8, wherein providing the trellis comprises 
20 optimizing a trellis for a particular transmission situation. 

12. A method according to any of claims 1-8, wherein providing the trellis comprises: 
generating a plurality of possible trellises; 

determining for each of the possible trellises one or more of a performance or a 
25 complexity parameter; and 

selecting one of said trellises responsive to said determining. 

13. An equalizer, comprising: 

a input channel for receiving received signals; 
30 a processor for applying a MD trellis methods in accordance with any of the preceding 

claims; and 



40 



WO 01/28174 



PCT/US00/28536 



an output channel for providing an estimate of symbols received on said input channel, 
based on said applying. 

14. A cellular telephone comprising an equalizer according to claim 13. 
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